Skip to content

feat(backend): Add TTL Strategy in Workflow Spec for Post-Completion Cleanup #11352

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 1 commit into from
Closed

feat(backend): Add TTL Strategy in Workflow Spec for Post-Completion Cleanup #11352

wants to merge 1 commit into from

Conversation

VaniHaripriya
Copy link
Contributor

@VaniHaripriya VaniHaripriya commented Nov 4, 2024 •
edited
Loading

Description of your changes:

  • Added the TTLStrategy field in Workflow Spec, configured to use SecondsAfterCompletion.
  • Uses the pipeline_ttlseconds variable to define the TTL duration.
  • This field automates cleanup of completed workflows.

**This PR should be merged only after #11758

Testing Instruxtions:

  • Build the API Server image and push to an image registry
  • Upload main.yaml file from here
  • Check in KFP UI Pipeline Spec tab if the following snippet is present:
platform_spec:
  platforms:
    kubernetes:
      pipelineConfig:
        pipelineTtl: 60
  • Create a run and check if TTL configuration is present in Workflow CR:
$ oc get workflow -o yaml $(oc get workflow --no-headers | awk '{print $1}') | yq .spec.ttlStrategy
secondsAfterCompletion: 60

Checklist:

@VaniHaripriya VaniHaripriya force-pushed the add-ttl branch 2 times, most recently from a0bf7c1 to 723aabd Compare November 4, 2024 21:15
DharmitD
DharmitD reviewed Nov 16, 2024
View reviewed changes
backend/src/apiserver/template/v2_template.go Outdated
@@ -77,9 +77,19 @@ func (t *V2Spec) ScheduledWorkflow(modelJob *model.Job) (*scheduledworkflow.Sche
}
}

var pipeline_options argocompiler.Options
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Camel casing required to stay consistent with golang variable definition standards - PipelineOptions

DharmitD
DharmitD reviewed Nov 16, 2024
View reviewed changes
backend/src/apiserver/template/v2_template.go Outdated
var obj interface{}
if util.CurrentExecutionType() == util.ArgoWorkflow {
obj, err = argocompiler.Compile(job, kubernetesSpec, nil)
obj, err = argocompiler.Compile(job, kubernetesSpec, &pipeline_options)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as https://github.com/kubeflow/pipelines/pull/11352/files#r1845247170

DharmitD
DharmitD reviewed Nov 16, 2024
View reviewed changes
backend/src/apiserver/template/v2_template.go Outdated
var obj interface{}
if util.CurrentExecutionType() == util.ArgoWorkflow {
obj, err = argocompiler.Compile(job, kubernetesSpec, nil)
obj, err = argocompiler.Compile(job, kubernetesSpec, pipeline_options)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as https://github.com/kubeflow/pipelines/pull/11352/files#r1845247170

DharmitD
DharmitD reviewed Nov 16, 2024
View reviewed changes
backend/src/v2/compiler/argocompiler/argo.go Outdated
}

const (
pipeline_default_ttlSeconds = int32(30)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Camel casing :)

DharmitD
DharmitD reviewed Nov 16, 2024
View reviewed changes
backend/src/v2/compiler/argocompiler/argo.go Outdated
@@ -86,6 +91,11 @@ func Compile(jobArg *pipelinespec.PipelineJob, kubernetesSpecArg *pipelinespec.S
}
}

pipeline_ttlseconds := pipeline_default_ttlSeconds
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Camel casing here too

HumairAK
HumairAK reviewed Nov 19, 2024
View reviewed changes
backend/src/apiserver/template/v2_template.go Outdated
@@ -300,9 +310,19 @@ func (t *V2Spec) RunWorkflow(modelRun *model.Run, options RunWorkflowOptions) (u
}
}

var pipeline_options *argocompiler.Options
for _, platform := range t.platformSpec.Platforms {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should be doing a nil check on Platforms cc @rimolive

@github-actions github-actions bot added the Stale label Jan 20, 2025
@github-actions github-actions bot closed this Feb 10, 2025
@rimolive
Copy link
Member

/reopen

@google-oss-prow google-oss-prow bot reopened this Mar 17, 2025
@google-oss-prow Google OSS Prow
Copy link

@rimolive: Reopened this PR.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@github-actions github-actions bot removed the Stale label Mar 18, 2025
@google-oss-prow google-oss-prow bot added size/L and removed size/M labels Mar 20, 2025
@google-oss-prow Google OSS Prow
Copy link

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign chensun for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@google-oss-prow google-oss-prow bot added size/M and removed size/L labels Mar 20, 2025
@VaniHaripriya
Add TTL Strategy in Workflow Spec
234db31 
Signed-off-by: VaniHaripriya <vmudadla@redhat.com>
@VaniHaripriya VaniHaripriya mentioned this pull request Mar 20, 2025
Closed
2 tasks
@rimolive rimolive mentioned this pull request Apr 9, 2025
Closed
2 tasks
@rimolive
Copy link
Member

rimolive commented Apr 9, 2025

/close

Closing in favor of #11802

@google-oss-prow google-oss-prow bot closed this Apr 9, 2025
@google-oss-prow Google OSS Prow
Copy link

@rimolive: Closed this PR.

In response to this:

/close

Closing in favor of #11802

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@HumairAK HumairAK HumairAK left review comments

@DharmitD DharmitD DharmitD left review comments

@hbelmiro hbelmiro Awaiting requested review from hbelmiro

Assignees
No one assigned
Labels
size/M
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@VaniHaripriya @rimolive @HumairAK @DharmitD